Displays and monitors are critical features of nearly every office, control center, purse, and pocket in the world as a means to transform data into information that can be readily received and understood by a viewer, user, or operator. Displays in recent years have transformed from a predominant role of passive information provider into a role that incorporates data input as well, providing an interactive experience as control technologies are added. The capacitive touch screen, for example, is ubiquitous for consumer electronics such as cellular telephones, tablets, and notebook computers, and is becoming commonplace on larger displays for control centers across government and industry, where operators now work long hours in front of touch screens instead of using legacy keyboard and mouse-type controllers.
Government and industry users have many displays and control centers that are in need of improved functionality, faster response time, and enhanced sensitivity to operator conditions. An immersive control environment enables operators of remote systems, sensors, manufacturing equipment, and unattended vehicles to perform at higher levels of effectiveness, provided operators are appropriately monitored for stress, overwork, fatigue, and other negative conditions. There is a large and rapidly growing need for technologies to improve the telepresence environment, and improve human performance and associated aspects of the human condition during teleoperation.
The opportunity is to provide an improved operator interface that provides the ability to use natural gestures, commands, and cues to provide real-world commands, but that also provides the ability for auxiliary monitoring systems to track, record, and assess operator performance and status. A system that performs both functions with a minimum of additional cost and training, and that augments existing control interface technologies will provide significant benefit to government and industry through reduced user fatigue, faster user response times, enhanced capabilities, and the ability to more easily monitor usage and performance without interfering with the interface itself.
Of particular value to these industries would be a subsystem that augments present touch screens, either during manufacturing or with an aftermarket overlay, enabling the use of laser pointers (or other highly collimated electromagnetic energy sources) to control the screen. In such a subsystem, a laser would enable the user to perform the traditional point, click, click-and-drag, multi-touch, and multi-touch-and-drag features known intrinsically by the majority of the population, except they would perform these actions with small motions of their fingers, wrists, or heads instead of with their arms. If the touch screen control software is sufficiently capable, laser pointer control could also provide additional “hotkey” functionality, with the addition of scroll functionality and assignable click functions using coded bursts of laser energy that can be transmitted much faster than a user could touch or multi-touch by hand. Additional functions could be assigned to gestures instead of clicks, providing a new category of commands that become relevant and possible for display input devices.
Remote optical control of displays in control centers would provide many advantages over conventional touch screen displays. Remote optical control would reduce operator fatigue, eliminating a common repetitive stress injury known as “gorilla arm,” a muscular and tendon soreness caused by large arm movements. It would improve response speed through multi-touch or multi-click actions, as well as large area movements such as a screen swipe or enlarge command that would be difficult to perform quickly by hand across a display having a large area, or even one the size of a large tablet or notebook computer.
An additional category of capabilities that are needed throughout the government and industrial control community, in particular, is the ability to monitor operators for signs of degrading performance, such as typically occurs under conditions of stress or fatigue. Optical control commands enable the use of infrared, ultraviolet, and other frequencies that are invisible to the human eye, so that fingers, hands, and/or head monitoring can be performed in a way that does not disrupt the operator. Tracking of infrared laser spots, for example, enables detection and assessment of subtle cues such as vibration, drift, degrading response time, and other performance characteristics indicative of fatigue, stress, drug use, and other human conditions of interest. These can be monitored and assessed with reduced processing complexity relative to facial video capture systems, and less obtrusively than electrical pad monitors and other skin sensor techniques.
Some attempts have been made to use remote controllers for display systems in the past. These have included infrared, microwave, and ultrasonic frequency emitters or detectors on the controller, with detectors or emitters, respectively, mounted to the sides of the display screen. Phase differences, timing differences, and other cues are used to determine the general direction of aim by the controller, and often other signaling techniques, including hardwired electrical switching, are used to determine the control action (e.g., “fire weapon” or “reload” for an arcade game). Controllers in the gaming community have also turned to sonic full-body assessment of large motions, providing capabilities for wider range variation and gesture recognition, but with very poor registration and aiming accuracy, and no capability to support multi-touch and other commands with high complexity and precision.
While these gaming systems are generally inexpensive to manufacture and install, and sometimes include reliable decades-old technologies, they suffer from significant limitations. All of these controllers have generally poor accuracy and precision when compared to modern touch screen displays. They do not generally incorporate multi-touch or multi-click features. They do not generally have the ability to upgrade the capabilities of an existing control room display or consumer device in an aftermarket capacity. All but the sonic field systems do not have a broad range of acceptable operating space, and typically can only properly register commands from tens of centimeters to a meter or two away. If the controller is closer or further, accuracy degrades further, if it functions at all, in terms of receiving triggering commands.
Developers have pursued integrating optical sensitivity of a screen or touch screen during its original manufacture. For example, the function of remotely controlling a touch panel using laser energy, where such sensitivity is added as part of manufacturing a touch panel explicitly with such capability has been attempted. In these situations, however, when any high power laser energy activates such a panel, they do not provide for the ability to employ separate targeting and actuating frequencies and energy levels, nor for multi-touch capabilities. Sensor and circuit elements providing this basic function, but without addressing the need for separate targeting and actuation, multi-touch capabilities, or aftermarket modification of pre-existing touch screens have also been described.
Despite the wide-ranging efforts by numerous developers to provide wireless or partly wired controllers for interacting with displayed information, no solution provides for upgrading a capacitive touch screen with an aftermarket capability to recognize direct, accurate spots for targeting. Furthermore, no solution provides for the capability to address multiple spots for multi-targeting or separate optical characteristics for spots used for operator targeting those used for triggering. No prior solution provides for multi-click and gesture recognition capabilities. No prior solution provides for separate unobtrusive operator monitoring capabilities at a low system and processing algorithm complexity. No prior solution addresses a wide range of display markets, from government control systems to consumer tablets with equivalent applicability.